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The present invention relates to software toots for testino and momtoring the operation of web^ased and 
other transsctiona) servers. 



A variety of comm^cially^avaflable software tools exist for assisting conpanies in testing the performence 
and functtonafity of their web-based transactional servers and assodated applications prior to deidoyment Eiamples 
of such tools mclude the LoadRuim®, WinRiinner® and Astra QuickTest® products of Mercury Interactive 
Corporation, the ass^nee of the present appl^tion. 

Using these products, a user can record or otherwise create a test script which specifies a sequence of user 
interactions with the transactional server. The user may also optionally specify certain expected responses from the 
transactional server, which may be added to the test script as verification points. For example, the user may record a 
session with a web4»8sed travel reservation system during wluch the user seenhes for a particular fG^ and may 
then define one or more verification points to cheek for an expected fiig^ 

Test scripu genmted through thb process are Idayed* or 'executed* to simulate the actions of users - 
tyincatly prior to deployment of tiie component being tested. During thb process, the testng tool monitors the 
p^orraance of the transactional server, indudifig determirnng the passifaO status of any verification points. Multiple 
test scripts may be replayed concorrentiy to simulate tiie load of a large ramiber of users. Using an automation 
interface of the LoadRunner product, it is possible to ifispatch test scripts to remote computers for execution. 

The results of the test are typicaBy communicated to tiie user through a series of reports that are accessible 
throui^ the user mterfaca of the testing tooL The reports may contein, for exanqde. graphs or charts of tha idiserved 
response tones for various types of transactions. P^ormance problems discovered timigh the testing process may 
be cormted by programmers or system admbnitrators. 

A variety of tools end serwces alsa exist tiist aiiow web site operators to mordtor the post-deployment 
performance of tiieir web sites. For examptoi Keynote Systems bic of San Mateo Caiffomia provides a service whMi 
uses automated agents to access a web site at regular intervels tiiroughout the day. The agents computersi* which are 
promded by Keynote Systems in sdected nuyor cities^ measure tiie tone requaed to perform various web site 
functions* and r^rt tim r^s to a savm proviM by Keynote Systems. The owner or operetor of tite web site can 
access this server usmg a web browser to view ti» coRect^ performance data on a city^rty or other basis. Other 
types of existir^ momtormg tools indude log analysis tools tiiat process sccess togs generated by web serverSr and 
packet sniffiog tools that monitor traffic to and from the web saver. 



Backnround of the hwentinn 
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Summary of the Invention 

A significant probtem with existing monitoring tools and services is that they often fail to detect problems 
that are dependent upon the attnliutes of typical end users, such as the user's locatwn* PC conf^uratron, ISP (Internet 
Service Provider), or bitemet router. For example* with some web site monitoring services, the web site operator can 
monitor the web site only from the agent computers and locations made available fay the service provider; as a result, 
the service may not detect a perf omiance problem seen by the most frequent users of the system (e.g., members of a 
customer service department who access the web site through a particular ISP« or who use a particular PC 
configuration). 

Even when such attribute-specific problems are detected, existing tools and services often fail to identify the 
specific attributes that give rise to the problem. For exemple, a mon'toring service may indicate that web site users in 
a particular chy are experiencing long delays, but may fail to reveal that the problem is experienced only by users that 
access the she through a parttcuiar router. Without such additional information, system administrBtois may not be 
abia to isalats and correct such problenis. 

Another significant proUem with existing tools and services b that they do not provide an adequate 
mechanism for monitoring the current status of the transactional server, end for promptly notifying system 
administrators whan a prpbtem occurs. For exanple, existing tools and services typicslly do not report a problem untH 
many minutes or hours after the problem has occurred. As a restdt, many end users may expOTonce the proUem 
before a system administrator beomues aware of the problem. 

The present invention addresses these end other problems by providing a software syst&n and method for 
monitoring the post-deptoyment operation of a web site system or other transectional server. In a preferred 
emboifiment the system indudes an agent component ragenti which simulates the actions of actual users of the 
transactionai swver while monitoring and reporting the server's performance, hi accordance with one aspect of the 
inventioni, tin agent ts sdapted to be installed on selected computers regent computers") to be used for monitoring, 
bidudlng computers of actual end users. For axanple, the agent couM be mstaUed on selected endwr computers 
witfun tiie various offices or organizations from which tiie transactionel server is comnmnly accessed. Once ti» agent 
component has been instaSed, tite agent computers can be remotely programmed (typically by the operator of tfia 
transactional server) using a controller con^onent rcontrolterl. The ability to flexibly select the computers to be used 
for moRitormg purposes, and to use actual end-user computers for monitoring, greatly facilftates the task of detecting 
probhuns associated witii tiw attributes of typical end users. 

bi accordance viritii anottier aspect of tiie invention, the controller provides a user Interface and various 
functions for a user to remotely select tiie agent computer(s) to include in a monitoring session, assign attributes to 
such computers (such as tiie toeatioiw organization, ISP and/or configuration of each computed, and assign 
transactions and execution schedules to such computers. The execution schedules may be periodic or repetitive 
sdiedulea; (e.g^ every hour, Monday tivough Friday), so tiiat the transactional server is monitored on a continuous or 
near*cominuous basis. The controller prefereMy represents tite monitoring session on the disiday screen es an 
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expanilable tree in which the transactions and execution schetfutes are represented as children pf the cerresponding 
computers. Once a monitoring session has been deffmed* the controller dispatches the transections and execution 
schedules to the respective agent computers over the faitemet or other network. The controller elso preferably includes 
functions for the user to record and edit transactions^ and to define aiert conditions for generating real-time alert 
S notifications. The controller may optionally be implemented as a hosted application on an Internet or mtraaet site; in 
which case users may be able to r^otely set up monitoring sessions using an ordinary web browser. 

During the monitoring sesstom. each agent computer executes its assigned transactions according to its 
assQned execution schedule, and generates performance data that indicates one or more characteristics of the 
transactional server's performance. The performance data may mchtdo; for example, the server response time and 

10 pau/faS status of each transaction execution event The passffai status values may be based on verification points 
(expected smer responses) that are defined within the transactions. The agent computers preferably report the 
performance data associated with a transaction immediately after transaction execution so that the performance data 
is available substantiaBy in real-tmie for viewing and generation of alert notifications. In tte preferred embodiment, 
the performence data generated by the various apnt computers is aggregated in a centralixed database which is 

IS remotely accessflile through a vireb-based reports server. The reports server provides various user-confqiurable charts 
and grapin that allow the operator of the transactional server to view the performance date associated with each 
transaction. 

In accordance with another aspect of the invention, the reports server generates reports whkh indicate the 
performance of the transactional server separately for the various operator-specified attributes. Using this feature, 

20 the user cam, for example, view and compare the performance of the transactional server as seen from different 
operator-spadfM hications (eig.. New York, San Francisco, and U.IU organixations (e.g., accounting, marketing, »id 
aistomei service departments^ ISPs |e.g.,Sprbig, AOL and EarthtM^^ The user may also have 

the option to f Bter out date associated with perticular attributes end/or transacthms (e.g.« exckide itota associated 
with AOL customers), and to deTme new attribute types (e.g., modem speed or operating system) for partitioning the 

25 performance data. The abity to monitor the performance data according to the operetor^spedf ied attributes greatly 
f acOitates the task of isotatmg and correcting attrftute-dependant performance problems. 

In Kcordance with another aspect of the invention, the performance data is monitored substantiaDy in real- 
tone (pref^bty by the ccntroSar) to check for any user defined alert conditions. When such an a)^ condition is 
detected, a notification message may be sent by Bna9, pager, or other communications method to an appropriate 

30 person. The alert conditions may optionally be specific to a particular hication, organization, ISP, or other ettribute. 
For example, a syston admMstrator respond for en Atlanta tonch office may request to be notified when a 
particulai: protriem (e.g., average response tima exceeds a particular threshokl) is detected by cominiters in that office, 
h the preferred emboifiment upon receiving an alert notification the adminto 
access the reports server and view the detaib of the event er events that tr^gered the n^ 
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In accordance with another aspact of the invention, the agent computers may be programmed to capture 
sequences of screen displays during transaction exacuthui, and to transmt these screen displays to the reports server 
for vtewing when a transaction fais. This feature allows the user to view the sequence of events, as 'seen* by an 
agent, that led to the error condition. 
S In accordence with another feature of the invention, an agent computer may be programmed to bunch a 

network monitor component when the path delay between the agent computer and the transactional server exceeds a 
preprogrammed threshold. Upon being launched, the network monitor component determines the delays currently 
being experienced along each s^ment of the network path. The measured segment delays are reported to personnel 
(preferably through the r^orts server), and may be used to detect various types of network problems. In accordance 
10 with enother aspect of the invention* one or more of the agent computers may be remotely progremmed to scan or 
crawl the monitored web site pmdicaiy to check for broken links (finks to ineccessftle objects). When broken links 
are detected, they may be reported by email through the reports server, or by other means. 

I S A distributed monitoring tool and essodated methods that embody the various inventhre features will now be 

described virith reference to the following drawings: 

Rgure 1 Blustrates the gmaal architecture of the monitoring tool, and Blustrates how the monitoring tool 
may be used to monitor the performance of e wdi-based transactional server. 

Rgure 2 illustrates a main user interface screen of the controller depicted in Figure 1 . 
20 Rgwes 3-9 illustrate the controller's Setup Wizard screens that are used to set monitoring sessions; 

Figures 10-12 ilwtrate screens of the controller's Alerts Wtzerd; 

Rgure 13-16 ilh»trate example status report web pages proidded by the we^ t.vnth 
Rgure 14 Dhistrating a representatwe *dri down' page returned when the user selects the drSI down link in Figure 1 3 
for the "browse order status* transacdoni 
25 Figures 17*18 are flow ifiagrams that iustrate the flow of information between components durii^ the 

setup end execution of a momtoring session. 

Figure 20 ifiustrates a process for capturing screen (fispiays on failed transactions. 

DetaBgd Descrtotion of the Preferred Fmbndgngnt 
The following description sets forth mmierous implementation-spedfic details of a distrdHited monitoring 
30 tool and associated methods. These detads are provided in order to illustrate a preferred enibodinram of the inventio 
and not to Emit the scope of the invention. The scope of the invention is defined only by the appended claims. 

Throughout this description^ it wil be assumed thet the transactional server beliig monitored is a web-based 
system that a accessible via the Imemet It will be recogirized, however, that the mventive methods can also be used 
to nnnitor other types of transactional servers^ bichiding those that use proprietary protocols or are accessible only to 
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internal users of a particular organization. For example, the underlying methodology can also be used to monitor 
internal intranets, two^ttar dient/server systenn. SAP Hf3 systems, an'd other types of distributed systems. 
L Overwewf 

Figure 1 illustrates the general components of the distributed monitoring tool, and illustrates how these 
components may be deployed wrthtn a network to test and monitor a web-based transactional server 30. Dashed lines 
in Rgure 1 indicate typical machine boundaries, with open boxes indicating one or- more maclnnes. As depicted by 
Figure 1. the transactional server 30 typically includes a web server component 30A and one or more appbcations 30B. 
The appfications may. for example; provide functionality for implementir^ one or more business processes, such as 
setting up a user account or placing an order. The applications 30B typically provide user access to one or more back- 
end databases (not shown). The transactional server may indude multiple machines, including machines that are 
geogrephically remote from one anoth^. 

As further depicted by Figure 1, the monitoring tool consists of three primary software components: an agent 
32« a coittroller 34 and a web-based reports server 36. Each component 32, 34. 36 Indudes one or more executable 
files or modules stored within a computer-readable mediun. 

The agent 32 uidudes the basic functionality for simulating the actions of users of the transactional server 
30 while monitoring and reporting server performance. As Illustrated fai Figure 1. the agent 32 is preferably iostaOed 
en multiple bitemet connected host computers 40 PCx wortotations, etc) so that the end user experience can be 
captured from multiple locatiims. These host computers 40 may advantageously include computers that are owned or 
controlled by the operator of the transactional server 30. For example, the operator of the transactional server can 
faistall the agwit component on sel^ted computers within each of the d^artments or organixations from wMch the 
transactional server is frequmtly accessed, including computers of actual end users. 

For convenience the computers 40 that host the agent 32 wiV be referred to as 'agent computers." and a 
conqniter 35 that hosts the contrdler 34 wiH be referred to as a 'controller computer.' It should be understood, 
however, that a singla computer could host two or more of the toofs components 32, 34, and 36, and that the 
functionarity of the monitoring tool couU be divhled differently between components, to addition, the. web reports 
senrn 36 and the trenssctiond server 30 could be eccessed through a common web d^^^ 

The controller 34 provides a user int^ace (UO throi^h whidi the operator of the transsctiond server can 
^ 19 and initiate monitoring sessions, inchiding distributed monitoring sessions in which the transactiond server is 
accessed end monitored from muhipte user locations. Throi^h this Ul, the user can. among other thing?, sdect the 
agent computers 40 to be induded within a monitorir^ session, and assign transactions and execution schedules to 
such computers. The controlter 34 dso provides functions for ^ncif ying dert comfitions, and for notifying pmynnd 
when such conditions exist Example screens of the controUar's Ul are shown in Figures 2-12 and 16 and are 

The wd> reporu server 36 provides functionality for dtowing the operetor to remotdy monitor the pisration 
of the transactiond server 30, as measured and reported by the agent computers 40, udng a standard web browser. 

-5- 
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In other embodiments, the repons server 36 could tie configured to 'push' the performance data, or reports generated 
therefrom, to a special client application for viewing. As described below, the agent computers 40 preferably report 
their transaction execution results (performance data) to the reports server 36 in real-time (preferably via the controler 
34, which checks for predefined alert conditions), aliowing operator to view the re^time status of the transactional 
server. The reports server 36 may optionally be imidemented by a "monitoring service provider* entity that stores and 
provides secure access to server status data for many different uansactionai servers and busmess entities; this 
approach refieves the operator of the transactional server under test from having to administer the reports server 36. 
Alternatively, each or some of the operators of the transactional servers under test could implement their own 
respective reports servers 36. 

As described below, one important feature of the monitoring tool involves the ability of the user to momtor 
server performance according to operator-sdected attributes of the agent computers 40. For example, using the 
reports server, 36, the user couM view a graph of the average response time as measured by all agent computers in 
San Fiandsco, or by aO computers that use a particular ISP. In one- emboifiment, the attranites of eecb agent 
computer indude the computer's location, orgai^ation, and ISP* and can be ass^ned or modified vie the user interfaca 
of the controller 34 (see 6). Other types of attributes, inchidmg user-defined attranite types, can additionally or 
altematively be used. An eiample of a report m which performance is displayed separately for each location and 
transaction is shown in Figure 1 5 and described below. 

Another nnportant feature invohres the abiTity of the user to assign execution schedutes to particidar agent 
irrachines 40, inchiding periMfic schedules (B.g., once per hour od weekdays). Using this feature, the user can, for 
exampler set up a monitoring session in which the transactional server 30 is proactively exercised and monitored on a 
contimious or near-continuous basis, and m which system administrators are notified immediately (such as by pager) as 
soonasao dert condition is det ected l 

To facilitate ao understamfing of the inventioo, the following tenninology wtN be used throughout the 
remakibig description! 

The term 'distributed meidtoring session' or 'distributed session* refers to a monitoiing session in which 
OMltlpie agent computers 40 are used to monitor a transactional server 30. 

The tOTi 'agent group* refers to the group of agent computers 40 included within a distributed sessbn. 

The tenn refers eith^ to the agent component 32 generdly, or to a particular copy or instance of the 
egent component running oo an agott compute, depending upon the context m which the term is used. 

The term *attr9iute' refers to a particular characteristic or property of a host or agent computer, such as the 
tocation, organization, IV. or conf^urathm of the computer. 

The term "imsactional server* refers to a multiuser system which responds to requests from users to 
perform one or more tasks or *tra]»actions^* such as viewmg account bif ormatioa piecing an order, perf omdng a 
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search, or viewing and sending electromc maiL The term ''operator' refers generally to a business entity that is 
responsible for the operation of the transactionai server (typically the ovvner). 

The term *testcase" refers generally to a computer representation of the transaction(s) to be performed by a 
particular computer to monitor a transactional server. In the preferred embodiment, the testcases indude conventional 
test scripts (either in textual or executabia form) that are 'played' by the agent computers 40^ although the tesicasas 
could alternatively be m other forms. Testcases may optionally include verification points that are used to test server 
functionality. 

The term 'web' indicates the use of the World Wide Web standards, such as HTTP. 
IIL Architecture and Cenaral Operation 

In a preferred embodiment the egent 32 is implemented using the commercially available LoadRunner Virtual 
User (VUserl component of Mercury Interactive Corporation, and is capaUe of executing testcases generated using 
Mercury Imerecthre's LoadRunner, WinRunner and Astra QuIckTest products. Examples of methods that may be used 
to generete end play testcases are described in co-pending U.& applications 08/949,680 IfBed October 14, 1997) and 
09f^7,448 (filed June 21, 1999L Other known programmbtg msthods for simulating user actions and monitormg 
server responses may be used to impleinent the agent 32; in additioa ^pfication-spedf ic hwdware couU be used to 
perform some or all of the egenfs functions. 

In the preferred embodiment, the agent 32 is fa»tall&l on the agent computers 40 prior to initiation of 
monitoring sessions. Once installed, the agent can recehre testcases and execution schedules from the controller 34 
over tiie Internet or otiier TCP/IP based network via API calls. Alternatively, the agents 32 may be bistalted 
automatically by the controller 34 when a monhoring session is initiated For example, the controller 34 could 
dispatch an agent 32 and a tastcasa (optkmaSy as a single executable component) to eech machina in the agent group, 
and the agmts 32 couki automatiealty delete themsehres folowiog testcase execution. Each agent 32 can preferably 
sbmdato tiie actions of multiple users. 

Preferably, tim agent group is selected so as to enoonvpass a representstive cross section of cfient attributes. 
For exarepte, one or more agent computers 40 may be selected wrttMi each geographic area and/or department from 
which significant user activity is expected to or^mate. 

In edditioa a monitoring service provicter entity, such as the entity that operates the r^orts server 36, may 
set up Internet hosts witij various attributes (ag., in various geographic locations, vwth a variety of different ISPs, 
etcj and make such hosts avadable to its customers as agent computers 40. Such host computers are pref^ly 
provhied by the service provkler with the agent 32 pre-installed, and are configured to monitor multiple transactional 
servers (end tinis service multiple opmton)concurremly. TIds metimd n especiafly useful where the opmtor of tim 
transactional server 30 woidd not otiierwise have access to client computers with attributes of typical end users. For 
exempt an operator of an electronic commerce Web site may nm have access to host co^ 
countries or regkms from which purchases are made. Themetiiodaborefevesthepperatorof tiie burden of setting up 
and adhndstermg tin agent computers 40. 

•7- 
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As Hlustrated in Figure 1« the controner 34 preferably inctudes or interoperates wttti a recorder 34A that 
provides functions for recording and editing transactions to be indudol within testcases. In a preferred embodkent, 
dny one or more of the above-mentioned products of Mercury Interactive Corporation may be used as the recorder 34. 
Rather than recordmg new testcases» the user may optionally re-use testcases or testcase scripts that were created 
for pro-deployment testing of the transactional server 36. Other existing tools end testcase generation methods could 
be used to generate the testcases. 

The controller 34 also includes a scheduler component 34B that is used to set up monitoring sessions. The 
scheduler 34B Is preferably implemented using one or more 'wizards' that step the user through the process of 
selectmg agent computers, specifying the transactions to be performed by such computers^ assigning execution 
schedules to the agent computers, and specifying criteria for generating alert events and notifications. Example screen 
displays provided by the scheduler 34B ere included in Figures 3-12 and discussed below. 

The controller 34 also includes en automation mterface 34C that provides metiuMls for controlfing tin 
operation of the agents 71 incbiding dispatching testcases and execution schedules to the agents. In a preferred 
embodiment, tiie automation interface is implemented using tiie LoadRunner 6.0 eutomation interface avaaable from 
Mercury Interactive Corgoratlon. The controRer 34 furtiier includes an alerts engine 340 tiiet monitors some or eB of 
the performance data generated by the egents 32 in real-time to check for user-def bied alert conditions. Usiiig tiie 
scheduler 348, tite elerts engme 340 can be configured to notify en operator of alert contltmns by en appropriate 
communications roetimd such as pager, ce&ular tetephoitt, or email For example, the alerts engine can be configured 
to page a system administrator whenever the average response time of the transactional server exceeds a certain 
threshol4 or when the transactional server becorras inaccessible from any bcation or organization. The al^s engine 
340 can also generate mitifications ti»t are based on tite content (e.g., expected text strings or values) returned by 
the transactional server. 

As depicted in Fqure h the controller 34 stores various test control data In local storage 38. The test 
control data typtcaliy bichides testcase files {script files end related data files) for prerecorded trensactions, and 
ses^ files tint specify the various inonrtoring sasskms tiist have been created^ 

As ouficated abov^ tim i^rts server 36 provides onKne^ web-based access to tiie testcase execution 

(performance) data reported in real-time by agents 32. As depicted in Rgure 1. tiie performance data for tiie ongoing 

(gstrtbuted sessions is stored within a central ''sessions" database 42, which is an OOBC compBant database in tfie 

preferred embodiment One possible schema of this database 40 is described below. As depicted by Figure 1, the 

components of tiie reporte server 38 preferably include a web server 36A such as Microsoft Internet Information 

Server {\fSl an accm control layer 368 which restricts access to the sessions database 42, a database access layer 

36C, and a repon generation componmit 360. The database access layer 36C is implemented using a set of Active 

Server Peges lASP files) tiiat use MOAC (Microsoft Data Access Components) to communicate with the sessions 

database 42. The ASP pages mchide an adimnistretion page (not shown) tiiat cen be eccessed by users witii 

administrator privily to perform sudi tasb as adding new end users to titt database 42^ 
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wo 01/16753 PCT/USOO/24303 
« 

Tha report generation component 36D is unpiemented using Microsoft ActiveX Data Olqects (ADO), which 
provides functions for generating dynamic web pages. The dynamic web pages inchides various pre^fined graphs 
and charts (see Figures 13>16) that are used to build customtied, web-based reports. The reports server 36 could Mso 
be configured to disseminate the reports by email, f ax« a push protocol, or other communications method. 
5 IV. Controller Ul and Session Setue 

Figure 2 illustrates the main screen or console for a preferred embodiment of the controller 34, with an 
example monitoring ses^on (also referred to as a 'profile') open and displayed in the tree window. The details of the 
monitoring session are graphically presented to the user as an expandable session tree 46 which shows the agent 
. (host) computers, the testcase execution schedutes assigned to each agent computer, and the transactrons assigned to 
1 0 each agent computer. The session tree also shows any alert conditions that have been defined. In the simple example 
shown in Rgura % the monitoriRg sesshin uses a single agent computer, "idopc* which has been ass^ned a sii^te 
transaction 'flights' and an execution schedule of "Every 5 minutes, Monday-Friday, all day.* The monitormg session 
inchides a smgte alert under which an alert event wril) be triggered if the response time of the transaction "ffights" 
exceeds 10 seconds. The expandaUe tree can advantageously be used to edit a monitoring session thm^ drag-and- 
15 drop and other standard functions provided by the Windows operating system. As illustrated in F^ure 16, the 
controSer's Ul also provides a browse window through which a user can view report pages from the r^uirts server 36. 

The controller's menu, the top level of which is shown in Figure 2, provides functions for performing various 
session-related tasks, inchidmg launching the Setup and Alerts Wizards (described below), opening and eifitmg an 
existing monitoring sessioiu starting and stopjung monitoring sessbns, specifying the address of the r^orts server 36 
20 to ba toed with a monitoriog sesshm, clearing the contents of the database 4^ and spaafying settings for semfing 
ann iiouiicAuunSi. 

To create a new monitoriog session^ the user selects PROFILEWEW, wMch causes the controller 34 to launch 
8 Setup Wizard (Figores 3-8). As ilhistrated by Figure 3, the user is imtially prompted to specify a session n The 
sesson name provides a mechanism for bter retrieving or viewmg the reports fo^ As 

25 afantrated in Ftgure 4, the user is then presented a "Select Transactmns* screen for specifyinp the previously- 
gmrated transactions to be included wtthm the monitoring sesshm. The user can also use the NEW button to launch 
the recorder 34A and record a new transaction. The transactions may optionally indude verification poims that specify 
expected swver responses, such as particular vahtes or text strings within web pages. AliemativBly. the transactions 
may stress the transactional server without verifying the content of the server responses. As described below, the 

30 user can later assign spedfic transactions, or sets of transactions, to specific agent computers 40, and can monitor 
the perf onnance of the transactional server on a transaction-by transaction basts. 

hi the preferred emboififflent, the user can freely define what constitutes a "transaction' for monitoring 
purposes. Forexaff9le,theusercanstart reconfog a user session, record any number of user interacthm^ 
server (form subnusdoniw page requests^ etcj, stop recording, and then store the result as a transaction under a user- 

35 specified name (OLg., "browse catatagli hi adiKtiorv Airing subsequent editing of the transaction, the user can 
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ppttonaKy divide the transaction into multiple smaOer uansactions or make other modifications. The transactions can 
also inchtde accesses to muttipte web «tes. Preferably, the transactions are defined by the user with suffictem 
granularity to fadlitate identification of performance bottlenecks. For example, the user may wish to create a 
separate transaction for each of the primary applications deployed on the transactu^nal server 30 so that each such 
application can be monitored independently. 

The transactions inchided within the session may optionally inchtde special non-destructive or "synthetic" 
transactions that do not change the state of the trensactional server 30. If destructive transactions are osed, the 
transactional server 30 may optionally be configured to handle such transaction in a special don-detnictive manner. 
This may be accompTished, for exampte, by setting up dummy accounts for momtortng purposes. In addition, where 
appropriete, the transactional server 30 may be preprogrammed to roll back its databases, or to otherwise ignore the 
transacttorv when a particular dummy account credit card nunib^, usemame, or other unique element is used. 

As Qhistrated by the "Select Conqtuters" screen in Figure S, the next step in the setup process invohres 
selecting the computer or con^titers to be included in the agent group. By selecting the ADD button from this screen, 
the user can view and select from B standard Windows NT® tree view of the hu^ 

hi one embodiment the tree view displays only those computers on which the agent 32 is mstaBed. In another 
embodirnent the tree view also Gsts computers that do not have the ege^^ 

for the user to rorotely install the agent on such computers. As indtcated above, the computes that are avalable for 
use may optitmally inchide computers that are made avaOable by a monitoring service provider; in such 
implementations, the Setup Wizard 34 may be configur«l to automatic^ly retrm a Hst of such service provider 

20 computers and their respective attrttHites frwn a special Internet host Techniques for genwating and accessing Osts 
of av^abie servers are wdl known ui the art and are therefore not ctescribed herein. The selected computers are 
added to the session tree 46 as rospecthm nodes or icons. 

When the user selects the EIMT button (Rguro 9 with a computer sebicted ffl 
presented with a tonqniter Propertio* screen as shown in F^re B. From tNs scieeit the user cen assign varbus 

2S anrBiutes (propertied to the computer or confirm previously^ in the ihistrated exaiiQde^ the 

attribute types are the location (e^g^ cityl organization C&g^ accountmg departmentl emi ISP of the agent computer 
40. Other prMfefmed attrHnites types that may be provided ihchide, for exempted a group nemo, the computer's 
iterating system, the router to which the computer is connected, the computer's modem or other connectnin speed, 
the computer's default web browser (particularty if the agent uses or emulates the browser^ and the hardware 

30 configuratmn of the computer. In additioa the controller 34 and the reports server 38 may provide the user an opthm 
to create one or more uttr-defined ettribute types, and to use such attribute types in the same manner as the pre- 
defined attributa types. It shouM be understood, therefore, that the specific attributes and attrSnites types shown in 
the figures are merely iustralhre, and are not mtended to limit the inventhm. 

The attributes tiiat are assi^ to the agent computers can be used to separately view the transactional 

35 servei^s performance as momtored by a partiodv anribute group (group of computers that share a particular attribute 
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or set of atviluites). For exiinipl«» the user can view a graph of the response times measured by all agent computers 
writh the location attribute 'San Jose' or the ISP attribute "Sprint/ Example reports are shown in Rguras 13-16 and 
are described below. The user can elso generate attribute-filtered reports to exclude performance data associated 
with specific attributes from consideration (as described below). The ability to v»w and monitor performance 
separately for each attribute group and to generate attribute-filtered reports greatly facilitates the task of identifying 
attribute-specific performance problems. 

When the user selects the NEXT button from the Select Computers screen, an "Assign Transactions' saeen 
(Figure 7) appears. From this screen, the usv can assign transactions (from the previously-created uansactions list) to 
spoafic computm in the agent group. The user can also specify, for each computer, the order in whmh that ccmtputer 
is to exKUte the assigned transactions. As transactions are assigned to agent computers 40, the transactions are 
added to the session tree 46 as chldren of their respective computers (as iOustrated in Figures 7 and 8 for the 
computer 'dolphin'). 

When the user selects the NEXT button from the Assign Transactums screen, an 'Ass^ Schedules' screen 
appears (Figure 8) that altows the user to assign a testcase execution schedule to each computer. When the user 
selects the SCHEDULE button with a computer selected in the session tree 46. a "Schedub Properties' box appears 
(FQwe 9). From the Schedule Pmperties box. the user can select a predefined execution schedule lB.f^ Nveekdaysl to 
assgn to the computer and/or define e new schedule. As illustrated in Figure 9, periodic schedules may be used. The 
pmKlic schedules may optionally include pseudo-random schedules. As shown in Figure 8, the schedules are added to 
tim session tree 46 as children of th»r respective egent computers, hi other embodiments, the schedules may be 
essigned on a trans^on-by-transaction bass. 

The execution schedules may be selected so as to prrnnde continuous or near-continuous monitonng of the 
transactional server 3(1 By staggering the execution schedides so that different agent con^uters 40 monitor the 
transactional server 30 at different times, tin transactional server 30 can optionally be monitored continuously (24 
hours per day) or neariy contnwousiy wttiioot using any single egent compute 40 for an extended period of tone. For 
example, H ti» agent computers 40 are ifotributed eniund tiie globe, tin sch^ 
computer 40 is used for testing during employee work hours Mdtiim its respective region. 

The Setup Wiiard may optionaSy pmvtde one or more hmctions (not Shistrated) for assistii^ users m setting 
up continuous or near-continuous monitoring sessions. For example, as tiie schedules are be'mg assiped to agent 
computera, tite wizard cotdd automaticdily detect and display the 'gaps' (periods of time difftng which tiie 
transactional server b not being numitored) in tiie cumulative execution schedule. The Setup VITaard could also provide 
an option to automaticaSy generate an execution schedule which fiUs-in tiiese gaps. In addition, a function couhf be 
provided for ensurbig tint at least two ^ent conqiuters 40 are scheduled to execute testcases at all times, so tiiat tiie 
hahire of a singte agent computer wiB not cause tiw transactional server to go unmonttored. 

When tiw user selects the FttttSH button (Figure 8) from tiie Assign Schedules box, tiia Setup Wixard doses 

and the user » presented with a view of tiie complete session tree 46. At this point, controller 34 dspatcbes tiie 
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testcases and eiecution schedules to the respective agent computers 40, and sends various session configuration data 
(session name, transaction identifiers, attributes of agent computerst etc) to the reports server 36 for storage in the 
sessions database 42. The controller 40 else stores a representation of the monitoring session in local storage 38. 
The generel flow of information to and from the controHer computer 35 is described below with reference to the data 
5 flow drawings of Figures 17 and 18. Once the setup process is completed, the monitoring sesmi contmues 
mdefimtety until hatted or terminated by the user. 

With the ses«on open within the controller's consoto (Rgure 2), the user can select ALEAT/ADD from the 
main menu to launch an Alerts Wizard (Rgures 10*12). As illustrated by Figure 10, the Alerts Wizard allows the user 
to specify one or moie performance parameters to monitor in real-time for purposes of generation alerts, inchidir^ 

10 response time» availability, pass/fail status, and response data size. By selecting the check box 70, the user can 
specify certain parameter statisttts to monitor, such as the average of the parameter over a specified time frame. 

As illustrated by Fqure 1 1 and 12, the Alerts Wizard also provides screens for specifying notification criteria 
for the parameters to be monitored. In the example shown In Figure 11, the user can request to be notiffed whenever 
the average response time exceeds a specified threshoidr or exceeds the threshold witii a specified frequency Ibjq^ 1 0 

IS times per minute). As shown in Figure ^Z the user cen elso request to be notified by pager or emei of an alert 
condition. 

The Alerts Wizard may also provide an option (not illustrated) to be notified when certain types of 
transactions fail andior when failures are detected within particular attribute groups. Using this option, a user can 
request to be notified whenever a problem is detected which faBs within tiie user's respective area of responsatility. 
20 For example, a system administrator responsible for a particular busmess process may be notifial when a transaction 
tint corresponds to that business process faBs; to avoid being notify of general failures, tills notification may ba 
made contingent upon other types of transactions compMng successfully. Otiier exemple uses of this feature inchide: 
notifying en ISP admiiustrator when a tiveshold number of egem computers u^ 

transactional server (optionelly contingent upon tiie transactional server being accessBile from otiier fSFtt and 
25 notifymg a system edmimstrator responsible for a particular office when a timhoM number of egent computers 40 

within that office em unable to access to tiie transactional server (optionally contingnit iqum tiie transactional server 

being wcesslble from otiter offices). 

bi otiier embci&n^ of the inv^ion, ti» various functions of the controller 34 could be implemented in- 
whole or iofart by tfte reporu i&m 36. For example, the above-described functions of tiie Alerts Wizanl and the 
30 associated functionafity of tiie alerts engine 340, could additionally or attematively be imi^emented by tiie reports 

server 36 such tiiat users can remotely set up and modify alert conditions. The task of checking for alarm conditions 

couM also be perfomied by ti» agents 32. 

Ni one enbodbnent tiie controller 34 is hosted by an ASP (application service provider) as a service that is 

accessed over tiie Intemet using a conventional web browser. Through tiie ASP'S servers; each customer b given 

35 secure access tote respective repository of testcase and session files. The service's user interfece for setting up 
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moftttoring sessions may be generally the same as shown in Rgures 2-12, with the dialog boxes replaced by 
corresponding web pages. Test scrqits (transactions) may be recorded using a server-side recorder, and/or may be 
recorded by the customer using a downloadable recorder and then uploaded to the server. The ASP, which may also 
operate the reports server 36 endlbr the agents computers 40, may charge customers for monitoring sessions based 
on one or more of the foHowing criteria, as well es others: number of transaction types monitoredL number of 
transaction execution events, quantity of hardware resources used, and time schedule and duration of monitoring 
sessions. One important benefit of opmting the controller 34 in this manner is that monitonng 'sessions can be 
initiated and modifiui from any computer that has Internet access, without the need for any special software. 
Another benefit is that the customer b relieved of the bunten have having to mstalt and maintain the controller 
software. 

In embodiments In which the controller 34 is hosted as a service, the task of asstgning execution schedules 
to the agent computers 40 may be performed by the ASP, rather than by the end user. This strategy is partttuMy 
useful where the agent computers 40 are shared by many different custoners, as it ^lows the ASP to distribute the 
load across the agoit computers so as to genereBy maxhnixe the total number of distributed monitoring ses«ons that 
can exist concurrently. A hybrid apjmiach is also possdile in which the customer controls the execution schedules of 
the customer's own agent computers 40 wlole the ASP controls the execution sctedules of the shared agent 
computer's that era under the ASrs control 

In yet other embodiments, the controller 34 may be hosted by a server on a imvate intranet, such as the 
intranet of the operator of the transactional server, in such conf ieurations, the controller preferably operates the same 
as if hosted by an ASP, but b private to the operator. 
V. Peyfiyro^ntftffffmr^ 

Rgures 13-15 ilhistrata Bxamples of the types of graphs or charts that ma^ 
% to facffitate remote monitoring of the transsciionet server 31 The exemples shown in Figures 13-15 Ohistrate a 
momtoring session tnvohnng fwe transactions: Mar Entry, Item in Stock Search, Browse Order Status, Update 
Account and Purchase from Stock. The transactions era being executed from egent computers 40 located in four 
get^rephic regions: New York, Japan, United Kingdom and San Francisco. More than one egent computer may be used 
in Mch such location. The names and granularities of the geographic locations can be defined by the operator during 
the setup process. 

The graphs mdtcate various aspects of tha transactbnal server's performance as monitored over a particular 
line frame (tha current day m this example). The first graph 76 (Figure 13) shows the mhmum, average, and 
manmom transaction times for each of the five transactions. The second graph 78 (Rgure 13) shows the average 
raqnnse tine for each transaction and each eno'hour intervat using a color coding scheme to distingwsh between the 
transactions. The third graph 80 (Figure 14 shows the distribution of service levels for ea^ 
nsing a irnkpie color for each level The fourth graph 82 shows, for each one^iour interval and each transaction^ the 
percentage of transactions that fa9edL 
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As illustrated in Figure 13, the report pages preferably include various Pnks and controls for allowing the user 
to generate customized and atuaiute-filtered views of the perfoiinance data. For example, if the user selects the "Ml 
down' fink for tiie 'browse order status* trmactkm, a page appears which includes the graphs 84, 86 shown in 
Figure 15. Both graphs 84, 86 shows aspects of the server response time for the Browse Orctor Status transaction 
broken down by locetion, as may be desirable to identify location dependent problems. The horizontal Bne hi these 
graphs 84. 88 represents a user-defined alert threshold. From this page, the user can driH down an adifitional level (by 
selecting the location-specific drili down finks 90) to view location-specific graphs for the Browse Order Stetus 
transaction. 

With further reference to Figures 13-15, the 'Report Parameters" window 88 aBows the user to moifify the 
time frame and/or the breakdown method used to generate the various graphs and charts. By modifying the 
breekdown method, the user can view the performance data separately for each transaction and for each attribute of 
the agent computers. In one embodiment, the perfomiance data can be viewed by transaction (shown in Hgures 13 
and 14L by location (shewn in Figure 15), by organizathm (not iDustrated), and by ISP (not Hlustratsd). In other 
embodiments, the performance date can be broken down accorifing to other attribute types, inehiding attribute types 
defined by the operator. 

The 'Bters* ofition 88 (Figures 13*15) alhsws the user to filter the displayed infonnation by transaction and 
by each of the attributes. Usmg this feature, the user can, for example, filter out from the reports the performance 
data corresponding to a particular transaction, bcation, organization, ISP, or combination thereof. bi one »nbo#nent 
{not shown), the us& specifies the f ater to be applied by completing a web form that includes a respective check box 
for each transaction and each attrilwte used in the monitoring session. The application of a filter, if any, is indicated 
by the notations at the tops of the graidis (e^f.. Transactions: AH; Locations: UK. NY; Organizations: accounting, 
marketing'l. 

The Graph List Ofrtion 80 allows the user to specify the set of default graphs that are to appear on the main 
status reporu page. The "Settings'* option 92 altows the user to eiSust and save other types of settings, such as an 
"auto refiash' rate (^g., every five minutes) end a starting date/time to be used whlun the reports. 

Figure 16 Hhistrates en exempla Transaction Healtb Distribution* chart that may be generated by the 
reports server 36. in thb example; the chart is bring viewed throu^ the browser window of the controlter's interface. 
The chart Is m the form of a Munensional matrix. The horizontal dimension represents the timeframe, can be 
modified by the user over a range of hours to years, bi this exan^, the cohimns represent hours of the current day 
(as displayed along the top of the ctott, and the rows r^iresent the transactions being monitored (as listed at the 
teft). The cells of the matrix are color-coded to reflect the response time of the particular transaction Airing in the 
particular time frame. Each hour «id each uansaction Is a hyperiink that, when selected, causes the view to change. 
For example if the user cl«ks on e particular hour, the timefreme changes to just that hour with the matrix's 
horizental dmnsion broken down into smaller (e.g., 5 or 10 mtnote) intervals. Simdarly, when the user clicks en a 
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transaction link« the vertical dimension changes so that the chart shows ordy the seldcted transaction, broken down 
according to geographical location (or possiily another computer attribute). 

As win be apparent from the foregoing examples, the ability to separately view and filter the performance 
data based on the attributes of the agent computers, inchiding operator-specified attributes, greetly simplifias the task 
S of identifying attr9)ut^$pedfic problems. Although spedfic attribute types are shown in the example reports, it shouU 
be understood that the iihistrated features can be andied to other types of attributes, inchiding user assigned attribute 
types. 

The reports server 36 also preferably proviites access to an Alerts chart (not shown) which contains 
infomiatton about the various alert events that have occurred. For each alert event, this chart may tndude, for 

1 0 example, an alert name; a color-coded mdication of the alert severity, the time of the alert event, the action taken (e.9.. 
'ernaa sent to admin9merc jntcom' or logged oRty"), and the text of any alert message sent. 
Vi. fNf H Pq?rt^ flmm 

The general flow of Information betvmn components during the setup and execution of e typical monitoring 
session will now be descrSied with reference to Figures 17-19. 

1 S Figure 1 7 illustrates the 'setup* or "programnmig* phase.of a monitoring session. As depicted by the left-to- 

right anrows in Figure 17, once the user completes the setup process, tha controller 34 dispatches the testcases 
(uansactions) and schedules to the respective agents 3Z Where the agents 32 reside on remote agent computers 40, 
the testcases vu! schedides are convminicated over the Internet using HTTP or another a TCP/IP based protocol via 
API caSs. As furth^ depicted by Figure 1 7, the controller also sends session configuration data to the reports smer 

20 36 (preferebly using HTTP) for storage in the sessions database 42. The configuration data includes the session name, 
identifiers ami properties (attributes) of the agent computers 40, and kf^ Where 
the reports server 38 services multipfe busioess entities, the configuration data may also include a usamame or other 
identifier of the bosmess entity to which the session corresponds. 

Table 1 summarizes, for one examphi end»odiment the tables that are created in the sessions database 42 

25 for each momtwkig session and used to generate the reports. Any of e variety of altemathre database schemes could 
be used. The various metrics that are displayed in the reports (e.g., average response time over a particular wmdow) 
are calculated using the data stored in the event meter table. 
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TABLE NAME 


DESCRIPTION 


Groups 


Contains tha names of ail ^ent computers and their associated properties. 


Transactions 


Contains a fisting of the transactions, by name, with each assigned a 
mimerical transaction ID. For each transactioa the table contains the 
thresholds used for evahiattng response times (e^^ less than 20 sec. - QIC 
from 20 to 30 sec - poor, etc). 


Status 


Contains a fisting of tha avallabia transacthm statuses (e.g., Pass^O, Fail- 1, 


Ranks 


Contauis a fisting of the threshold criteria names (e.g., 1*0K, 2-Waming, 
etc). 


Piroparties 


For each property defined by the user, a table is created that assigns a 
num^lcal ID to the set of members of that property (e.g^ for the 
'organizations' table might bichida the entrns R&Qo 1, Mark8tina-2. etc) 


bKtti Matar 


Contams tha results of each transaction execution event Each transaction 
execution event is represented by a record which contains the following data: 
record 10 pncreases sequentially with each new execution eventL transaction 
ID, result tstatus value), dateWme, response time in seconds, and properties 
of agent compute (location, organizatioa etc) 


Alarms Dafimtions 


Contains definitions of events that trigger alarms 


Alarms 


Stores a log of triggerad alarm conditions 



TABLE 1 - EXAMPLE DATABAISE SCHEMA 



As depicted by tha downward arrow in Figure 1 7, any alerts set up by the user are stored in local storage 38 
5 along whh session configuration data. The alerts may additionally or alternatively be communicated to the reports 
server 36. in which ease tha reports server may handle tha task of checking for and notifying users of alert comfitioiis. 

Figure 18 iihistrates tha ftow of data for a represemahve; remota agent 32 as tha agm 
During the execution procasSr the ageitt 32 interacts with (eg., sends HHP Post and Get messages to) the 
treosactional server 30 wh3e monitoring one or mora predefined performance parameters such as response time. Tha 

1 0 B8»it 32 also checks any v^ication pomts (ag., expected vahies or text streigs) defined within the testcese. Upon 
completing each transaction, the ^t 32 s»tds the resulting transaction execution data to the controll^ 34 using 
HTTP or another TCP/IP based protocoL The transaction execution data preferably includes a transaction ID, the 
performance data (response t&ne and passffail status) for the transaction, a transaction time/date stamp, and the host 
ID of the agent computer 40. The agents could alternatively be designed to report their execution on a more or less 

IS frequent basts (a^k once per server responsa, or once per testcase execution). The controller 34 compares tbe 
performance data to any predefined alert conditkms. Han alert condition b satisfied for whh^i a notification massage 
has been dtftned. the controller sends an alert notification massage (represented by a dashed fine in Fi^ 
eppropriate entity. Upon rec«ving an alert notification messagai the recipient can log into the reports server 36 to 
obtam details of the aim event such as tha location or orgamzatkm of tha agent computer that reported associated 

20 perfomiance data. The ^ events could also be stored locally to the controller computer and cfisplaye^ 
session tree 48. 
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As further dspicted by Fteure 18. the controller 34 forwards the transaction execution data and any satisfied 
atert conditions to the web reports server 36 (preferably using the HTTP protocol) for insertion into the sessions 
database 42. As with the agent-to-controlter communicetions, the controller preferably forwards the transaction 
eiecution data to the reports server 38 sidistantiafiy in real-time; on a transaction^y-transactmn basis. This is 
S accomplished in tin preferred embodiment through an API of the automation interface 34C (Figure 1). The aim events 
are detected and reported to the r^orts server 36 in real-time by the alerts engine 34D. If muhlpie agents 32 are 
scheduled to execute testcases concunrently, the controller 34 processes the data streams from tiie multiple agents 
CQOCunrently. The main controller loop is thus in the fonn of: 

10 wait for message from a Vuser (agent) 

route message to web reports server via AM call 

ApmAin jeportTransactionitiansaction, host, status, vahm) 
route message to alarms engnw 
go back to wait 

IS 

Vanous alternatives to the data flow process shown in Figure 18 are pos^. For example^ tiie agents 32 
could send the transaction executkm data directiy to the reports server 36, in which case the reports s&ver 30 could 
option^ forward some or aQ of tim execution data (e^, alert conditions only) to the controfier 34. In acUitioa all 
agent computers 40 witiiin a ghren location or organization couhf be configured to aggregate thar performance data for 
20 transmission to or rmriavai by the controler 34 or the reports server 38. In addition, the task of checking for and 
notifying users of alert con«tions could be performed by the agents 32 an«^ 

the comroSer 34. Further, tiie agents 32 couU be configured to 'filter' tha transaction execution data, so that only 
tfiose transactions that meet certain predefined criteria are reported. These and otiier eltematives coidd optionally ba 
prowM as user-conf^orable optionSi 

25 Rgure 19 iDustratss the proem of remotely accessing tiie sessions database 42 using a standard web 

browsw lOd As iostrated, tin user initiaily togs into his or her accoum using a usemamef^assword combmation or 
other autiientication mtJhtkL Thoeafter, the user views custoniiz«f, real-tima status reports (as described above) fc^ 
tiu transaction server or servers corresponding to that account As the reporu pages are requested, tite database 42 
is accessed and the various p^f ormance metrics calculated usmg programmmg metiiods tiiat are well known by titose 

30 sMIedintheart 

Vtt. Addttionat Features for Detectimi and Reflnrtimi ftahtons 

Three optional features for detecting and reporting error conditmns end performance problems wiH now be 
described. All tivee of tiiesa features are preferably implemented in part tiirougb executable code of tin egent 
component 32. 
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The first such feature involves having the agent computers 40 capture the screens returned by the 
transactional server 3) during transaction execution, ami then forward these screen captures to the reports server 36 
if the transactkm is unsuccessful When the end user drOls down on a failed transaction within a report, the reports 
server 36 displays, or presents an option to display, the captured sequence of screen ifispbys for the failed 
transaction. For example; if the f aHed transaction invohred an unexpected or mesing message on a web page, the user 
could view the entire web page as well as the web pages (inchiding any form data submitted by the agent) that 
preceded the unexpected response. An important benefit of this feature is the ability for the user to view the sequence 
of events that led to the failed transactkm. 

Figure 20 illustrates the screen capture process as implemented within the agent compon^it 32. As depicted 
by blocks 110*116, each time the agent 32 submits a request to the transactional ^er 30, the agent captures the 
screen returned by the transactional server and compares this response against any associated veriftcation points 
defuied within the transaction. The screen displays are preferably stored as bitmap images, but may aheroathrely be 
stored in another format such as HTML documents and associated objects. 

Once the transaction is fniished, the agent 32 determmes whether the transaction completed successfully. A 
trsnsaction is preferdrty tfsated as unsuccassfid if any verificetion point foiled. A transaction may also be treated as 
unsuccessful if, for exemple; a timeout event occurred. In the event of a transaction faihirB; ths agent 32 sends the 
sequence of captured screen disi||ays to the reports server 36 (block 1 20), which in turn stores the screen displays m 
the sessions database 42 for later vewing. The screen displays could additionaHy or alternatively be sent by emaa to 
a htfflian operator for viewing. If the transaction completes successfully, the screen ifisplays are discarded without 
being forwarded to the reports smer 3& 

A seamd feature that may be incorporated into the egent 32 is an abifity to measure aid report segment 
delays incurred ahmg 8 network path between en agent computer 40 and the transact Thesegment 
Mays are preferably measured usmg the Network Monitor component of the commercially-availabiB LoedRunner 6.0 
product of Mercury tnterective Corporation. Preferably, some or all of the agents 32 are confQured via the controHer 
34 to iaunefa the Network Monitor ton their respective agent contputers 40) when the path delay exceeds a 
preprogrammed thmdudd. These threshoMs may optionally be specified by the user when setting up a momtoifng 
sessloa Upon b«ng laimcheit the Network Monitor measures the delay along each segment of the path between the 
relevant agsnt computer 40 and the transxtkmal server 30 using wdl-known methods. The egent 32 then reports 
these measirements to the reports server 36, which ellows the user to driU down and view the measurements. The 
measured delays are preferably presentml using the standard segment delay and path delay graphs provided within 
LoadRunner 6.0. The segment delay data may be used, for example, to detect router problems or bottlenecks in 
network architectures. 

A third feature hnrohMs the ebBityd the agents 32 to detect and report ni^ links' Oinks to inacces^ 
fies or other obfects) within web siteaL this featura^ the user can remotely program one or more of the agent 
computers40 to crawl the web site periodicaSy |e.g., once per day) to check for broken links, and to report any broken 
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links that are found When broken links are located a list of the broken Gnks may automatically be posted to the 
reports server 36 for vievwng end/or be sent to the user by email, lech broken link may be reported to the user in the 
fomi of a URL pah^ that indicate the respective locations of the misstng ofa|ect and the page contahui^ the broken link. 
Techniques for crawling web sites and checking for Ivoken links are well known in the art, and are describeiL for 
example, in U.S. Patent No. 5,958,008 of Mercury Interective Corporation. As with other types of problems detected 
by the agents 32, when a particular olject is accessible from some agent computers 40 but not otiiers, tiie reports 
server 40 preferably aOows the user to separately view Hie attributes of the egent computers tiiat eiperienced the 
pnditem. 

Although the invention has been descnbed in terms of certain preferred embodiments, other embodtments 
that are apparent to those of ordinary skill in the art inchiding emboffimenu which do not provicfe all of the features 
and advantages set forth herein, are also wtthhi the scope of tilts invention. Accordingly, tiie scope of ti» invention is 
defined by the claims that foOow. in the method claims, reference characters are used for convenience of description 
only, and do not indicate a particular order for performing tiie metiiodl 



-19. 



wo 01/16753 



PCT/USOO/24303 



WHAT iS CLAIMED iS: 

I. A method for monitorino the performance of a deployed transaction^ server, comprising the steps 

of: 

(a) programming a phirality of computers to eiecute transactions on the deployed 
transactional server as sonutated users while morotoring performance of the transactional server; 

(b) assigning attributes to the' plurality of computers such that at least some of the 
computers have different attnbutes than others; 

(c) recehring and storing performance data generated by the phirality of computers as a 
result of step (aL the performance data imficating at least one parameter of the performance of the 
transactional server; and 

(d) (fisplaying the pi^omiance data separately for each of muhtple attributes assigned in 

steptb). 

Z The method as h Clatm 1, wherein st^ |d) comprises ganereting a report which separately 
nnScates the performance of the transsctionel server es moiutored from each of multiple geographic tocations. 

3. The method as in Clean 1« wherem step (d) comprises generatmg e report which separately 
indkatas the p«f ormance as monitored from each multiple organizations. 

4. The method as in Claim 1. wh^in step Cd) comprises generating a report which seperatety 
imficates the pvformance as monitored through each of multiple Internet service (voviders. 

5. The method as in Ctann 1, wh»etn step (a) comprises assigning automated execution schedules to 
the phirality of computers, wherein the execution scheddes spedf y timing for executing the Uansacthms. 

6. The method as m Clain 1, wherein step (a) comprises programming at least some of the plurality of 
computers to verify user-selected content of responses from the transactional server. 

7. The method as in Claim 1,wherein step (a) comprises progremmii^i et least some of the plurality of 
con^uters renNitely using a controller program that dispatches testcases t 

a The nttthod as in Claim 7, wherein the comroBer progrem is hosted on a server as a web-based 

senraoL 

9« The method as in Clatm 1, further comprising generating and transndttmg an alert message when 
the perfonnance data satisfies an alert condition. 

10. The method as in Claim 9. wherein the ^rt condition is specific to en attrauita-spectfic subset of 
the phirality of computm. 

I I. The method as in Claim h wherein step (c) comprises receiving and storing the perfonnance data 
substantially in reel time as trdnsactions are executed 

11 The method as in CIm 1, further comprising programming the transactional server to process a 
destructhm transaction executed by the one or more of the phveity of computers non-destructively. 
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1 3. The method as in Claim 1, wherein at least some of the plurality of computers are computers used 
by Old users of the transactional server. 

14. The method as in Claim 1, wbernn the performance data comprises e sequence of screen displays 
returned by the uansactionat server during a failed transaction. 

15. The method as in Claim 1« wherein the performance data comprises network segment delays 
measured by one or more of the plurality of computers. 

16. The method as in Claim 1, wherein the performance data comprises identifiers of broken web site 
links detected by one or more of the phirafity of computers. 

17. A computer-readable medium having a controller program stored thereon which, when executed by 
a computer, provlites functkms for at least: 

selecting a phirality of computers to inchide within a monitoring session for monitoring the post- 
deployment operation off a transactional server; 

assigiung transactions to the phiralhy of computers for exercising and monitoring the transactional 
server; and 

ass«mng attributes to specific computers of the phirelity of computers to permit monitoring of the 
transactional server separate for each of a plurayty of attributes. 

18. The computer-readable medium as in Claim 17, wherein the ettributes indude names of 
orgamiations in which the computers are located. 

ia The computer-readable medium as in Claan 1 1, wherein the attributes inchjde names of geographic 
regioos in which the computers ere tocated. 

2a The computer-readaUe medhm as in Cl^ 17. wherein the attributes include at least one of (a) 
bitainet tevte Piiwidefs used by the computers, emi (b) routes 

21. The computerHreadaUemediiDnes In Clakn 17, wherein th^ 
computers. 

22. The computer-readable medium as in Claim 17, wherebi the controller 
to essign ififf eroit transactions to iKffsrent computers of the plurafity. 

21 The comput^-read^le medium as in Claim 17, wherein the eontrolier program provides a function 
for assigntng execution schnhiles to the pbraKty of comput&s for executing the transactions. 

24. The computer-readable medhmi as in Claim 23, wherem the controller program provnles en optien 
for the usM" to itefine the execution schedules. 

25. The computer-readaUa medium as in dean 17, wherein the controll^ propam graphicafiy 
represents the momtoring session to the user as a tree in which the transactions are represented as children of 
corresponding computers. 

28. The computer-readaUe medium as in Daim 25, wherein the controller program further represents 
executkmschedides that are asslgnsd to the conquiters as respective didreninthetree. 
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27. The computsr-readable niedium as in Claim 1 7« in combinatwn with a raport server component that 
uses the assigned attributes to separate transactional server performance data for cfisplay. 

28. The computer-readabb medium as in Claim 17, in combination with an agent coinponnit which is 
adapted to be installed on the plurality of computers to prniit the computers to be remotely progremmed by the 
centroBar program. ^ 

29. The computer-readable m^um as in Claim V, wherein the controlter program provides a user 
option to specify an alert condition. 

30* The computer-readable medium as in Claim 29, wherein the controOer program provides an option 
to defme an alert condition that is specific to a subset of the plurality of attributes. 

3t. The computtf-readabte medium as in Claim 17, whenun the controller program implements 
functions for di^wtching the transactions to the phireiity of computeri 

32. The computer-ieadabte mediom as in Claim 17» wherein the controller progrem includes functions 
for recording the transecthms. 

33. The computer-readable medium es m Claim 17, wherein the controlter program inchtdes ftmctions 
for the user to ^wcify expected responses from the transactional server. 

34. A method for monitoring a dephiyed transactional server, comprising; 

remotely progranming e group of computers to access the delayed transactionat server as 
s'imutated users while monitoring perf onnance of the trensactional server such thst the transactibnai server 
ismonitofed substaityiy contiroiously by the group; 

monitoring perf or m a nce data generated by the group substantially m real-time to check for en alert 
iMMfitioiKend 

generating en alert rotification message in response to detection of the alert condition . 
XL The method as in Claim 34, wherein the elert condition is specific to en attribute^ased subset of 
can^uters of the group. 

38. The method as in Claim 34, further comprising storing the perf ormme data m a database 
sidistantially m re^time, and providing remote, onTme access to the database. 

37. The method as in Claim 36, wherein proviifing onfine eccess com{tfises providing a user option to 
specify e tbneframe over which to view the perf onnance data. 

38. The method as in Claim 34, wherein the group includes computers of endnisers of the transectional 

server. 

^. A method of monitorifig the op&ation of a transactional server, comprid'ng: 

nmaling an agent component on a phirality of computes, inchnfang computers of end users of the 
transactional server, wherain the agent component is edaptad to execute transitions as simulated users of 
the transacdonal senmr vvhile momtormg perfonnance of the transectional server 
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remotely asstgrnno to the plurality of computers the transactions to be executed by the agent 
components installed thereon. 

40. The method as In Claim 39* wherein assigning the transactions comprises selecting, on a computer- 
by-computer basts, the transactions to be ass^ned to each of the plurality of computers. 

41. The method es in Clebn 40» further comprising remotely assigning execution schedules to the 
phirality of computers for executing the transactions. 

42. The method as in Claim 41, wherein assigning the execution schedules comprises specifymg, on a 
computer-by-computer basis, the execution schedule to be used by each of the plurality of computers. 

- 43. The method as in Claim 39* further comprising aggregating and providing ronote access to 
perfomiance data coUected by the plurality of computers, the performance data indicating performance of the 
transactional server. 

44. The method as m Claim A3, wherein the performance data inchiites a sequence of screen displays 
captured by at least one of the plurality of computers during execution of a transaction. 

45. The method as in Claim 43, wherein the perfonnence data indudas network path segment delays 
measuied by at least one of the plurality of computers. 

48. The method as tn CWm 43, wherein the performance data irwiudes identifiers of broken web site 
finks detected by at least one of the plurality of computers. 

47. A method of monitoring operation of a transactional server, comprising, on each of a phirality of 
computers that are programn^ tomonit»' the transactional sarv^ as simulated users: 
. executmg a transaction as a simulated user of the transactional S9ver; 
Aving transaction executionr storing a so|uence of soms displays returned by the transactional 

serven 

de term inii ig whether the transaction compteted successfully; and 
. if the transaction cfid not complete successfully, forwanfiiQ the sequence of screen displays to a 
server for subsequent viewing. 
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